Skip to content

[GLUTEN-12436][VL] Map ORC files with all-_col* physical names by position even when orcUseColumnNames is true#12438

Open
beliefer wants to merge 1 commit into
apache:mainfrom
beliefer:12436
Open

[GLUTEN-12436][VL] Map ORC files with all-_col* physical names by position even when orcUseColumnNames is true#12438
beliefer wants to merge 1 commit into
apache:mainfrom
beliefer:12436

Conversation

@beliefer

@beliefer beliefer commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

What changes are proposed in this pull request?

Fixies #12436. This PR is related to facebookincubator/velox#18021.

Gluten decides the ORC column-mapping mode from a single global config:
spark.hadoop.orc.force.positional.evolution flips orcUseColumnNames, which selects
name-based vs position-based mapping for the whole query. This breaks any query that reads a
mix of ORC tables where some files carry real physical column names and some carry Hive
placeholder names (_col0, _col1, …):

  • With orcUseColumnNames=true (default): the _col* files have no real names to match, so every
    column reads back null → a filtered join side becomes empty and AQE folds the plan to
    LocalTableScan rows=0 (silently wrong result).
  • With orcUseColumnNames=false (positional): the real-name files are mapped by ordinal and land on
    the wrong column, failing with SCHEMA_MISMATCH (From Kind: INTEGER, To Kind: VARCHAR) or, when
    a string filter is pushed down, Filter(BytesValues, ...): testInt64Range() is not supported.

No single global value can read both tables correctly in one query.

Vanilla Spark makes this decision per file in OrcUtils.requestedColumnIds:

if (forcePositionalEvolution || orcFieldNames.forall(_.startsWith("_col"))) {
  // map physical schema -> data schema by INDEX
} else {
  // map by NAME
}

Gluten already honors the forcePositionalEvolution half (#12234). The missing half is
orcFieldNames.forall(_.startsWith("_col")): even in name mode, an ORC file whose physical schema
is entirely _col* must be mapped by position, because those placeholder names carry no identity —
the real names live only in the table schema. That per-file decision can only be made in the native
reader, which is the only layer that sees each file's physical field names.

This PR contains the Gluten-side change; the native-reader change is submitted separately to Velox
(facebookincubator/velox, see: facebookincubator/velox#18021), which adds the _col* detection to DwrfReader and falls back to positional mapping in name mode.

Gluten changeVeloxIteratorApi.setFileSchemaForLocalFiles:
Always attach the table (data) schema to the split for ORC/DWRF, instead of only when
orcUseColumnNames=false. The native reader needs the table schema in name mode too, so it can
remap the _col* file columns to the real names by position. Parquet behavior is unchanged (still
only attached when mapping by position). This is additive: the existing positional path and the
explicit orc.force.positional.evolution=true behavior are unaffected.

How was this patch tested?

Added a regression test to GlutenHiveSQLQuerySuite for all supported Spark shims
(spark33/34/35/40/41). The test:

  • Creates a Hive ORC table with columns literally named _col0/_col1 so the physical ORC field
    names are guaranteed to be placeholders (independent of the embedded Hive version; mirrors Spark's
    SPARK-34897 setup), plus a second metastore table with real names over the same files, and a
    separate real-name ORC table.
  • Without setting spark.hadoop.orc.force.positional.evolution (default orcUseColumnNames=true),
    asserts that the _col* table reads correct values via the real column names, that the real-name
    table still reads correctly by name in the same session, and that a join of the two returns a
    non-empty result (the original failure folded it to an empty LocalTableScan).
  • Confirms the scans stay native via checkOperatorMatch[HiveTableScanExecTransformer].

Note: the test passes end-to-end only together with the corresponding Velox change; the Gluten-side
change alone attaches the schema but relies on the native reader to perform the _col* positional
fallback.

Was this patch authored or co-authored using generative AI tooling?

co-authored with Claude Code (Claude Opus 4.8)

Copilot AI review requested due to automatic review settings July 3, 2026 07:20
@github-actions github-actions Bot added CORE works for Gluten Core VELOX labels Jul 3, 2026
@github-actions

github-actions Bot commented Jul 3, 2026

Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR addresses incorrect ORC reads in Velox when a single query touches a mix of ORC files with real physical column names and ORC files whose physical field names are Hive placeholders (_col0, _col1, …). It enables the native reader to make a per-file mapping decision (name vs position) by ensuring the requested data schema is always attached to ORC/DWRF splits, and adds a regression test across all supported Spark shims.

Changes:

  • Always attach the (requested) data schema to ORC/DWRF LocalFilesNode splits in VeloxIteratorApi, even when orcUseColumnNames=true.
  • Add a regression test (spark33/34/35/40/41) covering mixed _col*-schema ORC files + named-schema ORC files in the same session/query.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 6 comments.

Show a summary per file
File Description
backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxIteratorApi.scala Always attaches schema for ORC/DWRF splits so native reader can perform per-file mapping decisions.
gluten-ut/spark33/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveSQLQuerySuite.scala Adds regression coverage for _col* placeholder ORC files + named ORC files in one session.
gluten-ut/spark34/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveSQLQuerySuite.scala Same regression test for Spark 3.4 shim.
gluten-ut/spark35/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveSQLQuerySuite.scala Same regression test for Spark 3.5 shim.
gluten-ut/spark40/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveSQLQuerySuite.scala Same regression test for Spark 4.0 shim.
gluten-ut/spark41/src/test/scala/org/apache/spark/sql/hive/execution/GlutenHiveSQLQuerySuite.scala Same regression test for Spark 4.1 shim.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

val joined = sql(
"select c.name, n.label from test_orc_colstar_renamed c " +
"join test_orc_named n on c.id = n.uid")
checkAnswer(joined, Seq(Row("a", "b")))
s"create table test_orc_named(uid int, label string) " +
s"stored as orc location '$namedLoc'")
hiveClient.runSqlHive("insert into test_orc_named select 7, 'b'")

val joined = sql(
"select c.name, n.label from test_orc_colstar_renamed c " +
"join test_orc_named n on c.id = n.uid")
checkAnswer(joined, Seq(Row("a", "b")))
val joined = sql(
"select c.name, n.label from test_orc_colstar_renamed c " +
"join test_orc_named n on c.id = n.uid")
checkAnswer(joined, Seq(Row("a", "b")))
val joined = sql(
"select c.name, n.label from test_orc_colstar_renamed c " +
"join test_orc_named n on c.id = n.uid")
checkAnswer(joined, Seq(Row("a", "b")))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CORE works for Gluten Core VELOX

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants